Method and apparatus for identification

ABSTRACT

A method and system for de-identifying a video sequence are provided. The method may include the steps of capturing a video sequence, comprising a number of individual frames, including one or more users performing one or more actions, and using activity recognition to recognize one of the one or more actions. One or more of the plurality of frames may be defined as comprising the recognized one or more actions, and a portion of the one or more of the plurality of frames may be identified to remain visible. The non-identified portions of the one or more of the plurality of frames and the non-defined frames may be de-identified. This method may be applied to the determination of whether a user has ingested a medication pill.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/595,441, filed May 15, 2017 to Hanina et al., titled “Identificationand De-Identification Within A Video Sequence”, which is a continuationof U.S. patent application Ser. No. 14/990,389, filed Jan. 7, 2016 toHanina et al., now U.S. Pat. No. 9,652,665, issued May 16, 2017, titled“Identification and De-Identification Within a Video Sequence”, which isa continuation of U.S. patent application Ser. No. 13/674,209, filedNov. 12, 2012 to Hanina et al., now U.S. Pat. No. 9,256,776, issued Feb.9, 2016, titled “Method and Apparatus for Identification”, which, inturn, claims the benefit of U.S. Provisional Patent Application Ser. No.61/582,969, filed Jan. 4, 2012 to Hanina et al., titled “Method andApparatus for Identification.” The contents of all of the priorapplications are incorporated herein by reference in their entirety.

This application incorporates by reference the entire contents of thefollowing applications, and any applications to which they claimpriority, or otherwise incorporate by reference:

Method and Apparatus for Verification of Medication AdministrationAdherence, Ser. No. 12/620,686, filed Nov. 18, 2009 to Hanina et al.

Method and Apparatus for Verification of Clinical Trial Adherence, Ser.No. 12/646,383, filed Dec. 23, 2009 to Hanina et al.

Method and Apparatus for Management of Clinical Trials, Ser. No.12/646,603, filed Dec. 23, 2009 to Hanina et al.

Apparatus and Method for Collection of Protocol Adherence Data, Ser. No.12/728,721, filed Mar. 22, 2010 to Hanina et al.

Apparatus and Method for Recognition of Patient Activities whenObtaining Protocol Adherence Data, Ser. No. 12/815,037, filed Jun. 14,2010, which claims the benefit of Apparatus and Method for Recognitionof Patient Activities When Obtaining Protocol Adherence Data, U.S.Provisional Patent Application 61/331,872, filed May 6, 2010 to Haninaet al.

Apparatus and Method for Assisting Monitoring of Medication Adherence,Ser. No. 12/899,510, filed Oct. 6, 2010 to Hanina et al.

Apparatus and Method for Object Confirmation and Tracking, Ser. No.12/898,338, filed Oct. 5, 2010 to Hanina et al.

Method and Apparatus for Monitoring Medication Adherence, Ser. No.13/189,518, filed Jul. 12, 2011 to Hanina et al., which claims thebenefit of Method and Apparatus for Monitoring Medication Adherence,61/495,415, filed Jun. 10, 2011 to Hanina et al.

FIELD

This invention relates generally to the monitoring of patient medicationadherence to a prescribed regimen, patient behavior, medical procedureor other patient or healthcare provider activity, and more particularlyto obscuring the identity of a video recording of a patient, whileallowing for adherence to a medication protocol, or adherence to otherdesired patient activity to be confirmed.

BACKGROUND

Determination of adherence of patients to a medication protocol isdifficult. While direct observation may be employed, it may be expensiveand inconvenient. Watching medication administration over a videoconference may be employed, but is also expensive and inconvenient inthat both the patent and an administrator must be on a video or otherconference call at the same time. Finally, the inventors of the presentinvention have determined that these conventional systems fail toprotect the identification and privacy of patients and patient data,while still allowing for determination patient activity.

SUMMARY

Therefore, in accordance with one or more embodiments of the invention,video sequences of patients administering medication may be recorded.The video sequences may be preferably de-identified in a manner toobscure patient identifying information while still allowing for areviewer (either computerized or human) to determine proper medicationadministration from the video sequence. Additional embodiments of theinvention may apply to activities performed by a healthcare provider, orother assistant, thus allowing for confirmation of proper action bythem, while maintaining the privacy of the patient identification.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification anddrawings.

The invention accordingly comprises the several steps and the relationof one or more of such steps with respect to each of the others, and theapparatus embodying features of construction, combinations of elementsand arrangement of parts that are adapted to affect such steps, all asexemplified in the following detailed disclosure, and the scope of theinvention will be indicated in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the invention, reference is made tothe following description and accompanying drawings, in which:

FIG. 1 is a flowchart diagram in which video frames are de-identifiedafter transmission to a remote location in accordance with an embodimentof the invention;

FIG. 2 is a flowchart diagram including object and activity recognitionin which video frames are de-identified before transmission to a remotelocation in accordance with an additional embodiment of the invention;

FIG. 3 is a flowchart diagram in which video frames are de-identifiedbefore transmission to a remote location in accordance with anadditional embodiment of the invention;

FIG. 4 is a flowchart diagram in which video frames are de-identifiedbefore and assigned a unique identifier in accordance with an additionalembodiment of the invention;

FIG. 5 depicts a data transmission relationship between a video capturedevice and a remote storage and computing location in accordance with anembodiment of the invention;

and

FIG. 6 depicts details of both the video capture device and remotestorage and computing location of FIG. 5.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Storing a video sequence of a patient or other individual takingmedication to be later used to determine adherence to a particularmedication protocol may comprise a difficulty as storage of such patientrelated information may run afoul of HIPAA or other patient privacylaws.

Therefore, in accordance with one or more embodiments of the invention,one or more methods or apparatuses depicted in one or more of the aboveapplications incorporated herein by reference may be employed in orderto record one or more video medication administration sequences by auser. These sequences are preferably de-identified before, upon or afterstorage, or before, upon or after transmission. The method ofde-identification in a first embodiment of the invention includesblurring of various facial features of the patient so that the patientidentity cannot be determined from the de-identified video sequences,but maintains an unblurred portion of the patient where the medicationis to be administered Other de-identification methods may be employed,such as rotoscoping, substitution of an avatar or other cartoon-likecharacter, or the like. If may also be desirable to overlay actualportions of a video sequence, such as an inhaler, blood pressure cuff,injectable medication administration device, and the like. It iscontemplated in accordance with one or more embodiments of the inventionthat maintaining or providing such an unblurred portion of the imageonly be provided for one or a few number of frames determined to be at acritical time for determining patient medication adherence. Furthermore,an indicator, such as a border, highlight or the like may be provided toemphasize these unblurred portions. Other portions of the image may beblurred that are not in the region of interest, i.e. background or otherareas of the image not necessary to determine proper medicationadministration.

In the case of taking a pill orally, in order to better protect theidentity of a patient, it may be preferable to only unblurr the pill inthe mouth region of the image, rather than the entire mouth region ofthe user. Alternatively, it is possible that the patient's mouth andsurrounding area is preferably not blurred, thus providing a widerviewing area of the image. Furthermore, as noted above, it is possibleto implement this unblurring process in only key frames for determiningmedication administration, or on all frames as desired. Such unblurringmay be implemented when, for example, an object of interest isrecognized in a frame, such as when it has been determined that a pillis correctly positioned on the screen, or the pill has been determinedto have been placed in the mouth of the user. Such de-identification mayalso employ facial averaging, or any other technique that may beemployed to remove patient identifiable information while still allowingfor a determination of proper medication administration. Any ofde-identification techniques noted in any of the above applicationsincorporated herein by reference may also be employed. The patient maybe shown a de-identified version of their video images on a display ofan image acquisition device, thus confirming to them that their privacyis being protected. Furthermore, the process may be applied to otherprocedures, such as surgery in a hospital, other hospital of healthcareprovider scenarios, in medical education, or the like.

Furthermore, one or more preliminary steps, such as identification of amedication pill or the like, may not be blurred, as long as doing sowill not compromise the identity of the patient, such as if the pill isin front of the face of the patient. In this case, other views of thepill may be unblurred, thus allowing for identification while preservingthe secrecy of the identity of the patient.

The blurred video sequences may further be employed in an automatedactivity recognition sequence employing computer vision to determinewhether the patient has properly administered the medication, or can bereviewed manually by a user to determine adherence to the protocol. Acombination of these two methods may also be employed.

Referring first to FIG. 1, a method for employing the present inventionin which video frames are de-identified after transmission to a remotelocation is depicted. As is shown in FIG. 1, at step 110, facialrecognition or other identity confirmation may be employed. The identityof the patient may be confirmed through machine vision. Such facialrecognition may be provided for differentiation between multiple users,or to simply confirm that a single person is the correct person.Additionally, such facial recognition may be employed to identify userssequentially, such as in the case of identifying a healthcare providerand then a patient. The face, hands, fingers, other body parts, orgestures being made by a particular body part of the patient may betracked through the screen to allow for continuous identification ofposition determination, if desired. Alternatively, one or more biometricmarkers, password or other acceptable identification techniques may beemployed. Then at step 120, object detection may be employed to confirmproper medication is being employed. Next, at step 130 activityrecognition may be employed in order to determine whether the patienthas performed a particular desired step or steps, preferably related tomedication administration or ingestion. Machine learning and computervision may be employed in this step. This activity recognition step maybe performed before step 120, in parallel with step 130, or after step130.

Next, a number of relevant video frames may be selected for transmissionat step 140. Thus, frames comprising a number of frames comprising thoseneeded for object and activity recognition may be selected fortransmission, including a predetermined number of leading and followingframes. Such selection may be employed to reduce the amount of data tobe transmitted and possibly reviewed. Of course, the entire recordedvideo sequence may also be transmitted. Such selection may be performedin an automated manner, such as a predetermined number of frames beforeand after some critical object, activity recognition, recognition ofsuspicious behavior, predetermined frames in relation to one or moreinstruction prompts, UI guides, or the like. In such a case, selectionof video frames to be reviewed may be performed upon receipt oftransmitted frames at the remote location, or all of the frames may bestored at the remote location. It should also be understood that suchtransmission is intended to include transmission to a remote storagedevice, such as a remote computer network, database, cloud storage orthe like, and may also comprise transmission to a storage device on alocal area network, via the Internet, or to a locally attached storagedevice. Such transmission may also take place from a mobile device toany of the remote locations described above, and may comprise anyappropriate apparatus for acquiring and processing data, and thentransmitting the processed data.

Once received after transmission, image segmentation may be performed atstep 150 to segment the background and other portions of the image fromthe patient and medication, and may be employed to desegment the mouthand medication area from the rest of the image, if desired. Thissegmentation step may be skipped, if desired, or may only be applied toa subset of the transmitted images. Thus, such segmentation processingmay only be applied to particular frames based upon proximity to item orgesture recognition, or in some other manner as noted above. Thesegmented patient images (or unsegmented images) may then be subject tode-identification processing at step 160, as described above, to removethe ability to identify the patient from the images, while retaining theability to view the video sequence and confirm proper medicationadministration. Portions of one or more of the images may not bede-identified in a manner in accordance with the process as describedabove. Patient faces may be tracked through the various video imagesequences to continue to confirm identity, to link face with otherbiometric identity confirmation to be sure the patient does not leavethe screen, and may be used to confirm proximity of the patient to thescreen, if desired, to allow for proper viewing of administration at alater time, by either automated or manual review. As an alternative inall embodiments, tracking or object recognition of the mouth portion andmedication pill portion of the image may be employed, allowing or theblurring of the remainder of the image determined to be other than thetracked mouth and/or medication pill, and such partial blurring may beapplied only in selected frames, as described above.

After such de-identification, a unique user identifier may be assignedto the de-identified video sequence at step 170 so that any user mayview the video information without learning the identification of thepatient, thus preferably complying with one or more desired privacyschemes. This identifier may be related to a particular patient,particular image capture device, or the like. The patient will thereforebe un-identifiable from the de-identified video image sequence, but thesequence will be attributable and pinned to a particular patient record,thus allowing for proper correlation by one or more healthcare providersable to access such identification information.

Referring next to FIG. 2, a method for employing the present inventionin which video frames are de-identified before transmission to a remotelocation is depicted. As is shown in FIG. 2, at step 210, facialrecognition or other identity confirmation may be employed. The identityof the patient may be confirmed through machine vision. Alternatively,one or more biometric markers, password or other acceptableidentification techniques may be employed. Then at step 220, objectdetection may be employed to confirm proper medication is beingemployed. Next, at step 230 activity recognition may be employed inorder to determine whether the patient has performed a particulardesired step or steps, preferably related to medication administrationor ingestion. Machine learning and computer vision may be employed inthis step, in any known manner, such as those set forth in one or moreof the above applications incorporated herein by reference. Thisactivity recognition step may be performed before step 220, in parallelwith step 230, or after step 130.

Image segmentation may next be performed at step 240 to segment thebackground and other portions of the image from the patient andmedication, and may be employed to desegment the mouth and medicationarea from the rest of the image, if desired. This segmentation step maybe skipped, if desired, or may only be applied to a subset of thetransmitted images. Thus, such segmentation processing may only beapplied to particular frames based upon proximity to item or gesturerecognition, or in some other manner as noted above. Facial or otheridentification processing may be employed at this time to confirm theidentity of the user. Other identification methods may also be used,either alone or in combination, including, but not limited to,fingerprint recognition, voice recognition, or password. Afterdetermining such identity, the segmented (or unsegmented) patient imagesmay then be subject to de-identification processing at step 250, asdescribed above, to remove the ability to identify the patient from theimages, while retaining the ability to view the video sequence andconfirm proper medication administration and ingestion. Portions of oneor more images may not be de-identified in a manner in accordance withthe process as described above. As an alternative in all embodiments,tracking or object recognition of the mouth portion and medication pillportion of the image may be employed, allowing or the blurring of theremainder of the image determined to be other than the tracked mouthand/or medication pill. Such partial blurring may also be only appliedin one or more selected frames, in accordance with the process notedabove. Tracking of other body portions may be employed for anyparticular desired test, such as tracking the arm of a user to be sure ablood pressure cuff is properly used, or to determine whether othertesting (such a drug or urine testing) has been properly performed.

After such de-identification, a unique user identifier may be assignedto the de-identified video sequence at step 260 so that any user mayview the video information without learning the identification of thepatient, thus preferably complying with one or more desired privacyschemes. This identifier may be related to a particular patient,particular image capture device, or the like.

Next, a number of relevant video frames may be selected for transmissionat step 270. Thus, frames comprising a number of frames comprising thoseneeded for object and activity recognition may be selected fortransmission, including a predetermined number of leading and followingframes. Such selection may be employed to reduce the amount of data tobe transmitted and possibly reviewed. Of course, the entire recordedvideo sequence may also be transmitted. Such selection may be performedin an automated manner, such as a predetermined number of frames beforeand after some critical object, activity recognition, recognition ofsuspicious behavior, predetermined frames in relation to one or moreinstruction prompts, UI guides, or the like. In such a case, selectionof video frames to be reviewed may be performed upon receipt oftransmitted frames at the remote location, or all of the frames may bestored at the remote location. It should also be understood that suchtransmission is intended to include transmission to a remote storagedevice, such as a remote computer network, database, cloud storage orthe like, and may also comprise transmission to a storage device on alocal area network, via the Internet, or to a locally attached storagedevice. Such transmission may also take place from a mobile device toany of the remote locations described above, and may comprise anyappropriate apparatus for acquiring and processing data, and thentransmitting the processed data. This frame selection step 270 may alsobe performed before the image segmentation and de-identification steps250 and 260 if desired in order to reduce computational powerrequirements.

Referring next to FIG. 3, a method for employing the present inventionin which video frames are de-identified before transmission to a remotelocation is depicted. As is shown in FIG. 3, at step 310, facialrecognition or other identity confirmation may be employed. The identityof the patient may be confirmed through machine vision, or through othermethods, such as analyzing the biometrics of a person, including face,fingerprint, palm print, voice and the like. Alternatively, one or morebiometric markers, password or other acceptable identificationtechniques may be employed. Image segmentation may next be performed atstep 320 to segment the background and other portions of the image fromthe patient and medication, and may be employed to desegment the mouthand medication area from the rest of the image, if desired. Thissegmentation step may be skipped, if desired, or may be applied to asubset of the transmitted images. Thus, such segmentation processing mayonly be applied to particular frames based upon proximity to aparticular determination of item or gesture recognition, or in someother manner as noted above. The segmented (or unsegmented) patientimages may then be subject to de-identification processing at step 330,as described above, to remove the ability to identify the patient fromthe images, while retaining the ability to view the video sequence andconfirm proper medication administration and/or ingestion, inhalation,injection or the like as appropriate. Portions of one or more of theimages may not be de-identified in a manner in accordance with theprocess as described above. As an alternative in all embodiments,tracking or object recognition of the mouth portion and medication pillportion of the image may be employed, allowing or the blurring of theremainder of the image determined to be other than the tracked mouthand/or medication pill. And such partial blurring may be applied in onlyselected frames, as described above. After such de-identification, aunique user identifier may be assigned to the de-identified videosequence at step 340 so that any user may view the video informationwithout learning the identification of the patient, thus preferablycomplying with one or more desired privacy schemes. This identifier maybe related to a particular patient, particular image capture device, orthe like. The patient will therefore be unidentifiable from thede-identified video image sequence, yet the sequence will beattributable and pinned to a particular patient.

Next, a number of relevant video frames may be selected for transmissionat step 350, or alternatively, this step may be performed before imagesegmentation and de-identification steps 320 and 330, if desired, inorder to reduce computational power requirements. Thus, framescomprising a number of frames comprising those needed for object andactivity recognition may be selected for transmission, including apredetermined number of leading and following frames. Such selection maybe employed to reduce the amount of data to be transmitted and possiblyreviewed. Of course, the entire recorded video sequence may also betransmitted. Such selection may be performed in an automated manner,such as a predetermined number of frames before and after some criticalobject, activity recognition, recognition of suspicious behavior,predetermined frames in relation to one or more instruction prompts, UIguides, or the like. In such a case, selection of video frames to bereviewed may be performed upon receipt of transmitted frames at theremote location, or all of the frames may be stored at the remotelocation. It should also be understood that such transmission isintended to include transmission to a remote storage device, such as aremote computer network, database, cloud storage or the like, and mayalso comprise transmission to a storage device on a local area network,via the Internet, or to a locally attached storage device. Suchtransmission may also take place from a mobile device to any of theremote locations described above, and may comprise any appropriateapparatus for acquiring and processing data, and then transmitting theprocessed data.

Referring next to FIG. 4, a method for employing the present inventionin which video frames are de-identified before transmission to a remotelocation is depicted. As is shown in FIG. 4, at step 410, identityconfirmation may be employed. The identity of the patient may beconfirmed through machine vision, facial recognition or the like.Alternatively, one or more biometric markers, password or otheracceptable identification techniques may be employed. Acquired patientimages may then be subject to de-identification processing at step 420,as described above, to remove the ability to identify the patient fromthe images, while retaining the ability to view the video sequence andconfirm proper medication administration. Portions of one or more of theimages may not be de-identified in a manner in accordance with theprocess as described above. Such partial blurring may be applied only inselected frames, as described above.

After such de-identification, a unique user identifier may be assignedto the de-identified video sequence at step 430 so that any user mayview the video information without learning the identification of thepatient, thus preferably complying with one or more desired privacyschemes. This identifier may be related to a particular patient,particular image capture device, or the like. A unique identifier mayalso be provided for the medication being administered by the patient,if desired. Such a medication may be identified using gesturerecognition, object detection or the like.

Each of the above-described embodiments of the invention allows rapidreview and streaming of a video sequence acquired of a patientadministering medication from a server along with time, date, location,unique identifier, dose taken, medication name, and other patientactivities that have been automatically logged, while maintaining theprivacy of the patent identity and any other confidential or privatepatient information. Such rapid review may include the automated (ormanual) designation of a portion of the total sequence of images thatmay or may not include a non-de-identified portion for review by aviewer. Thus, a predetermined number of frames before and after anevent, such as before and after object detection of a medication, orgesture or activity recognition of ingestion or other medicationadministration, may be designated. As such, only these designated framesneed be shown to the user, thus resulting in a substantially reduced setof images for review by a viewer. This may allow for a rapid review of alarge number of medication administration sequences by the viewer. Ifthe full video sequence is desired to be provided, it is contemplated inaccordance with an alternative embodiment of the invention that theviewer be provided with the ability to skip or “fast forward” the videosequence to each of one or more groups of designated “important” frames,preferably displaying non-de-identified video portions, and preferablybeing designated as noted above in accordance with object, activity orother recognition. A slider or other time or location selection devicemay also be provided to allow a viewer to quickly and easily review andselect various portions of a video sequence. Annotation of the videosequence may also be provided to allow for notes to be saved for furtherreview, for example.

In order to further aid the viewer, it is contemplated in accordancewith one or more various embodiments of the invention that one or moreportions of one or more of the non-de-identified frames, or of thede-identified frames, be highlighted, zoomed, or otherwise amplified orhighlighted in order to allow for a more precise review thereof by aviewer. This will allow the viewer to better determine suspiciousbehavior, proper administration and ingestion, or the like. Suchhighlighting may be performed in an automated manner based upon suchrecognition, or may be provided in a manual manner, such as indicatingan item to be highlighted by an operator. Once highlighted, thehighlighted area may be tracked through a sequence of frames so that theobject, such as a medication, can be followed through a video sequence.

Various embodiments of the present invention may further employ facetracking that would track the user, eliminate other faces or otherdistracting or possibly identifying items in the video sequences. Suchtracking may be employed in real-time, near-real time, or after storageand or transmission of the video sequence data. Upon use of such facetracking, it is contemplated in accordance with embodiments of theinvention that if the face of the patient leaves the screen, the entirescreen may be blurred, or otherwise de-identified. Furthermore, if theface of the patient turns sideways or to another difficult to see angle,the entire screen may be blurred. Whether the face or other object, evenif completely blurred, tracking technologies may still be employed toconfirm location of the patient face, object, or the like.

Such processing may be applicable to medication administration, and anyother patient procedures, including in hospital, doctor office,outpatient, home or other settings. Thus, in accordance with variousembodiments of the invention, administration of any item may bedetermined, while maintaining privacy of the user, by blurring orotherwise de-identifying video image sequences, while maintainingnon-blurred video portions including the administration procedure.

In accordance with various embodiments of the invention, referring nextto FIG. 5, a remote information capture apparatus 500 is shown. Suchapparatus is adapted to allow for the capture and processing ofinformation in order to implement the system and method in accordancewith the applications noted above and incorporated herein by reference.Such information capture apparatus 500 is placed in communication with aremote data and computing location 700 via a communication system 600,preferably the Internet or other communication system, such as wirelesscommunication, wireless LAN, or any other communication systemappropriate for transmitting video and other data. Via communicationsystem 600, information captured by apparatus 500 is transmitted toremote data and computing location 700, and analysis information orother instructions may be provided from remote data and computinglocation 700 to apparatus 500. In accordance with a preferred embodimentof the invention, any data to be transmitted may first be encrypted onapparatus 500 and unencrypted at remote data and computing location 700.Indeed, it is contemplated in accordance with one or more of the notedembodiments of the invention that encryption of recorded information,either before or after de-identification thereof, will improve securityof patient identification and robustness of the system in general.

It is further contemplated that a plurality such information captureapparatuses 500 may be coordinated to monitor a larger space than aspace that can be covered by a single such apparatus. Thus, theapparatuses can be made aware of the presence of the other apparatuses,and may operate by transmitting all information to one of theapparatuses 500, or these apparatuses may each independently communicatewith remote data and computing location, which is adapted to piecetogether the various information received from the plurality of devices500. Each such apparatus 500 may comprise a mobile computing device,such as a smart phone or the like including a web camera, or a laptopcomputer or pad computing device, each including a web camera or thelike. Therefore, in accordance with one or more embodiments of theinvention, Processing of video information may be performed locally,remotely, or partially locally and remotely. Thus, de-identification andobject and/or activity recognition may proceed locally, while frameselection may proceed remotely. Any other combination of locally andremote processing may also be provided, also including encryption anddecryption schemes. Furthermore, and local remote, or combination ofdevices may each process a portion of the captured information, and maythen transmit processed information to a single location for assembly.Finally, encryption of video may be performed locally, and then afterdecryption, all processing described above may be performed. Such remotelocation may comprise a cloud location, remote computer or group ofcomputers, a remote processor or multiple processors, and the like, andmay further include one or more attached or further remote storagedevices for storing computer, computer program and/or video information.

Referring next to FIG. 6, a more detailed view of a preferred embodimentof remote information capture apparatus 500 and remote data andcomputing location 700 is depicted. As is shown in FIG. 6, apparatus 500comprises an information capture device 510 for capturing video andaudio data as desired. A motion detector 515 or other appropriatetrigger device may be provided associated with capture device 510 toallow for the initiation and completion of data capture. Informationcapture device 510 may comprise a visual data capture device, or may beprovided with an infrared, night vision, or other appropriateinformation capture device. A storage location 520 may be furtherprovided for storing captured information, and a processor 530 isprovided to control such capture and storage, as well as other functionsassociated with the operation of remote information capture apparatus500. An analysis module 535 is provided in accordance with processor 530to perform a portion of analysis of any captured information at theremote information capture apparatus 500, in the manner as describedabove. Apparatus 500 is further provided with a display 540, and a datatransmission and receipt system 550 and 560 for displaying information,and for communicating with remote data and computing location 700.Remote data and computing location 700 preferably comprises systemmanagement functions 730, and a transmission and reception system 750and 760 for communicating with apparatus 500. Such system managementfunctions comprise, at least in part, many of the centralized functionsnoted in any of the pending patent applications noted above andpreviously incorporated herein by reference, or any of the abovedescribed functions. Transmission and reception system 750 and 760 mayfurther comprise various GPS modules so that a location of the devicecan be determined at any time, and may further allow for a message to besent to one or more individual apparatuses, broadcast to all apparatusesmeeting one or more determined requirements, or being used foradministration of a particular prescription regimen, or broadcast to allavailable apparatuses. Remote location also may include computerprocessors, storage, etc. appropriate for performing all functionsdescribed above being performed at such a remote location, aftertransmission of video and other data from a local computing device.

It has been further determined by the inventors of the present inventionthat when employing object and activity detection to determine, forexample, the identity of a medication, and more particularly when themedication pill has been placed in the mouth of a user, it may bedifficult to determine the location of the medication pill, and thusconfirm proper medication ingestion. This is because, in the case of alight colored or white medication pill, for example, the teeth of theuser may look similar to the pill. Therefore in accordance with anembodiment of the invention, after (or instead of) a pill candidate hasbeen identified during a step determining medication ingestion, a nextstep of processing may be performed to confirm that a mouth colorsurrounds the pill candidate. Thus, if the pill candidate turns out tobe the tooth of a user, to the right and left of the tooth candidatewill be other teeth, viewed as similarly colored by the video camera.Thus, by confirming that to the right and left, top and bottom of a pillcandidate is mouth colored, it is possible to determine that the pillcandidate is a single entity, and is therefore in the mouth of the user.Thus, by using a recognition system for recognizing a pill candidatesurrounded by a mouth of a user, rather than simply looking for amedication pill, robustness of the object and activity recognition canbe enhanced. This process may also be applied to isolation of a tonguepiercing, for example. If not possible to remove such a piercing, theuser may be asked to remove the piercing before using the system.

Similarly, when a user is moving a pill while holding it in their hand,it is possible to use the combination of the pill and fingertips of theuser to differentiate the medication pill from other background itemsthat may appear similar to a pill. Thus, through the tracking of motionof the medication pill through an expected path of travel, andsegmentation of not only the medication pill, but also thefingertip/pill/fingertip combination, improved object and activityrecognition may be achieved. In such a manner, in accordance with anadditional embodiment of the invention, first, one or more objects maybe detected and presented as pill candidates. Then, from this set ofpill candidates, further segmentation may be performed to confirmwhether the pill candidate is held between two finger tips to confirmand differentiate that the pill candidate is in fact a medication pilland not some other object. Of course, the determination may simply lookfor a pill/fingertip combination from the outset.

Furthermore, because individuals may each hold a pill differently, inaccordance with yet another embodiment of the invention, two or moredetection methods may preferably be used. Thus, in this particularembodiment, for example, in a first frame one may search for an oblongmedication pill being held horizontally between fingertips, and in asecond frame for the oblong medication pill being held verticallybetween fingertips. Of course, and number of different objects may besearched for, and in this example, orientation of the medication pill at45 or 135 degrees may also be employed. Furthermore, these multiplemethods may be employed on each frame, depending on processor speed.Employing these methods on different frames provides a more real-timeanalysis with a slower processor, for example. Furthermore, with the useof a multi-core processor, the different methods may be processed inparallel on the different cores without sacrificing the real timeeffect.

Instructional feedback may be provided to the user to aid in properpositioning. This real time feedback makes the user aware that thetracking system is in operation, and may reduce the likelihood that theuser will try to trick the system. By searching for these variousdifferent types of images in sequential or different frames, thelikelihood of properly identifying the medication pill, regardless ofhow it is handled, is greatly increased. Of course, rather than simplylooking for differently oriented images, one could, for example, lookfor color in one or more frames, shape in one or more frames, markingsin one or more frames, and any other type of identifier in similargroups of one or more frames. Thus, any of these recognition systems maybe employed, essentially in parallel. Furthermore, the use of such aplurality of methods of detection may allow for increased confirmationof a correct medication pill being identified. Additionally, once themedication pill has been identified, it is possible to select the oneattribute generating the highest confidence score, and then use this oneattribute to continue to track the medication pill through subsequentframes of the video images.

As one goal of the present invention is to confirm proper medicationadherence and ingestion of medication, small movements, micro audiosounds suggesting swallowing, or the like, of the user may be employedin order to further confirm proper medication administration, reducingthe likelihood of a user tricking the system. So, for example,identification of any swallowing motions in the neck or throat, gulletmovement, jaw movement, or audio information related to swallowing, suchas sounds related to the swallowing of water or the like, may be furtheremployed in order to lend additional confirmation to the ingestion of amedication pill. Therefore, in accordance with one or more embodimentsof the present invention, teaching the system to recognize such micromovements may be employed in a manner similar to the teaching of thesystem to recognize the medication pill. These movements may becorrelated in time to properly determine sequence of administration, andmay be compared to the first use by the user of the system in acontrolled environment, such as a clinic. Additionally, all of thefeatures of the invention noted above may be applied to such micromovements, including non-de-identifying these micro movements, whenrecognized, in all or a subset of the frames of captured video asdescribed above. It may be desirable to de-identify any micro audiosounds, such as through sound synthesis, audio mix, sampling or the liketo further protect the identity of the user.

Furthermore, magnification of these movements may be provided so thatmanual subsequent review of these portions of the video sequences may beeasier. Additionally, during video capture, these portions of the usermay be zoomed, to appear larger in the frame, and allowing for an easierautomated and manual determination of such micro movements. Thecombination of automatic recognition and/or manual review of these micromovements may be used in conjunction with any of the methods forconfirming medication administration and ingestion noted within thisapplication in order to improve the confidence with which medicationadministration and ingestion may be confirmed.

It will thus be seen that the objects set forth above, among those madeapparent from the preceding description, are efficiently attained and,because certain changes may be made in carrying out the above method andin the construction(s) set forth without departing from the spirit andscope of the invention, it is intended that all matter contained in theabove description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

It is also to be understood that this description is intended to coverall of the generic and specific features of the invention hereindescribed and all statements of the scope of the invention which, as amatter of language, might be said to fall there between.

What is claimed:
 1. A method for de-identifying a video sequence,comprising the steps of: capturing a video sequence, comprising aplurality of individual frames, including one or more users performingone or more actions; using activity recognition to recognize one of theone or more actions; defining one or more of the plurality of frames ascomprising the recognized one or more actions; identifying a portion ofthe one or more of the plurality of frames to remain visible; andde-identifying the non-identified portions of the one or more of theplurality of frames and the non-defined frames.
 2. The method of claim1, wherein the de-identifying comprises blurring the image.
 3. Themethod of claim 1, further comprising the step of highlighting thevisible portions of the one or more plurality of frames.
 4. The methodof claim 1, wherein the one or more actions comprises ingestion of amedication.
 5. The method of claim 1, further comprising the step oftransmitting the de-identified one or more of the plurality of frames toa remote storage and computing location.
 6. The method of claim 1,wherein the step of using activity recognition to recognize one of theone or more actions further comprises the steps of: determining, throughobject recognition, the identity of an object for a first plurality offrames of the video sequence; and determining, through objectrecognition, the identity of an object and its surroundings in a secondplurality of frames of the video sequence.
 7. The method of claim 1,wherein the one or more actions comprise micro movements of one or morebody parts.
 8. The method of claim 1, wherein the step of using activityrecognition to recognize one of the one or more actions furthercomprises the steps of: defining a plurality of detection methods forrecognizing one of the one or more actions; and applying each of theplurality of detection methods to a subset of the frames of the capturedvideo sequence.
 9. The method of claim 8, wherein each of the pluralityof detection methods comprise searching for an object positioned at adifferent orientation.
 10. A method for de-identifying a video sequenceof a user ingesting a medication pill, comprising the steps of:capturing a video sequence, comprising a plurality of individual frames,including one or more users performing the action of putting themedication pill in their mouth; using activity recognition to recognizethe placement of the medication pill in the mouth of the user; definingone or more of the plurality of frames as comprising the recognizedaction of placing the medication pill in the mouth of the user;identifying a portion of the one or more of the plurality of frames toremain visible corresponding to the portion of the one or more of theplurality of frames corresponding to the location of placing themedication pill in the mouth of the user; and de-identifying thenon-identified portions of the one or more of the plurality of framesand the non-defined frames so that the portion of the one or more of theplurality of frames corresponding to the location of placing themedication pill in the mouth of the user remains visible.
 11. The methodof claim 10, wherein the de-identifying comprises blurring the image.12. The method of claim 10, further comprising the step of highlightingthe visible portions of the one or more plurality of frames.
 13. Themethod of claim 10, further comprising the step of transmitting thede-identified one or more of the plurality of frames to a remote storageand computing location.
 14. The method of claim 10, wherein the step ofusing activity recognition to recognize one of the one or more actionsfurther comprises the steps of: determining, through object recognition,the identity of an object for a first plurality of frames of the videosequence; and determining, through object recognition, the identity ofan object and its surroundings in a second plurality of frames of thevideo sequence.
 15. The method of claim 10, further comprising the stepsof capturing a video sequence, comprising a plurality of individualframes, including one or more users performing the action of swallowingthe medication pill; using activity recognition to recognize one or moremicro movements of one or more body parts associated with the swallowingof the medication pill by the user; defining one or more of theplurality of frames as comprising the recognized one or more micromovements of the one or more body parts of the user; identifying aportion of the one or more of the plurality of frames to remain visiblecorresponding to the portion of the one or more of the plurality offrames corresponding to one or more micro movements of the one or morebody parts of the user; and de-identifying the non-identified portionsof the one or more of the plurality of frames and the non-defined framesso that the portion of the one or more of the plurality of framescorresponding to one or more micro movements of the one or more bodyparts the of the user remains visible.
 16. The method of claim 8,wherein the step of using activity recognition to recognize theplacement of the medication pill in the mouth of the user furthercomprises the steps of: defining a plurality of detection methods forrecognizing the placement of the medication pill in the mouth of theuser; and applying each of the plurality of detection methods to asubset of the frames of the captured video sequence.
 17. The method ofclaim 16, wherein each of the plurality of detection methods comprisesearching for a medication pill positioned in the mouth of the user at adifferent orientation.
 18. A system for de-identifying a video sequenceof a user ingesting a medication pill, comprising the steps of: a videocapture device for capturing a video sequence, comprising a plurality ofindividual frames, including one or more users performing the action ofputting the medication pill in their mouth; a processor associated withthe video capture device using activity recognition to recognize theplacement of the medication pill in the mouth by determining, throughobject recognition, the identity of an object for a first plurality offrames of the video sequence, and determining, through objectrecognition, the identity of an object and its surroundings in a secondplurality of frames of the video sequence, defining one or more of theplurality of frames as comprising the recognized action of placing themedication pill in the mouth of the user, identifying a portion of theone or more of the plurality of frames to remain visible correspondingto the portion of the one or more of the plurality of framescorresponding to the location of placing the medication pill in themouth of the user, and de-identifying the non-identified portions of theone or more of the plurality of frames and the non-defined frames sothat the portion of the one or more of the plurality of framescorresponding to the location of placing the medication pill in themouth of the user remains visible.
 19. The system of claim 18, whereinthe de-identifying comprises blurring the image.
 20. The system of claim18, wherein the processor further applies each of a plurality ofdetection methods for recognizing the placement of the medication pillin the mouth of the user to a subset of the frames of the captured videosequence.